System for external fish parasite monitoring in aquaculture

ABSTRACT

A system for external fish parasites monitoring in aquaculture, the system comprising: —a camera ( 52 ) suitable to be submerged in a sea pen suitable for containing comprising fish, the camera being arranged for capturing images of the fish; and—an electronic image processing system ( 86 ) configured for identifying external fish parasites on the fish by analyzing the captured images, wherein: —a ranging detector ( 54 ) configured for detecting the presence of fish and measuring a distance from the detector to the fish is mounted adjacent to the camera ( 52 ); and—an electronic control system ( 12 ) is arranged to control a focus of the camera ( 52 ) on the basis of the measured distance and to trigger the camera ( 52 ) when a fish has been detected within a predetermined distance range.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national stage entry under 35 U.S.C. § 371 ofPCT/EP2018/085740, filed on Dec. 19, 2018, which claims priority to U.S.Ser. No. 62/608,411, filed on Dec. 20, 2017, and EP18154085.7, filed onJan. 30, 2018, the content of PCT/EP2018/085740 is hereby incorporatedby reference in its entirety.

TECHNICAL FIELD

The invention relates to a system for external fish parasites, such assea lice, monitoring in aquaculture, the system comprising:

-   -   a camera suitable to be submerged in a sea pen suitable for        comprising fish, the camera being arranged for capturing images        of the fish; and    -   an electronic image processing system configured for identifying        external fish parasites, such as sea lice on the fish by        analyzing the captured images.

In this specification, the term “monitoring” designates any activitythat aims at providing an empirical basis for a decision whether or nota given population of fish is infested with external parasites. The termmonitoring may also include a way of determining to which extent a fishis infested with external parasites. Although the monitoring may becombined with measures for destroying or killing the parasites, the termmonitoring in itself does not include such measures.

BACKGROUND

Like humans and other animals, fish suffer from diseases and parasites.Parasites can be internal (endoparasites) or external (ectoparasites).Fish gills are the preferred habitat of many external fish parasites,attached to the gill but living out of it. The most common aremonogeneans and certain groups of parasitic copepods, which can beextremely numerous. Other external fish parasites found on gills areleeches and, in seawater, larvae of gnathiid isopods. Isopod fishparasites are mostly external and feed on blood. The larvae of theGnathiidae family and adult cymothoidids have piercing and suckingmouthparts and clawed limbs adapted for clinging onto their hosts.Cymothoa exigua is a parasite of various marine fish. It causes thetongue of the fish to atrophy and takes its place in what is believed tobe the first instance discovered of a parasite functionally replacing ahost structure in animals. Among the most common external fish parasitesare the so called sea lice.

Sea lice are small, parasitic crustaceans (family Caligidae) that feedon the mucus, tissue, and blood of marine fish. A sea louse (plural sealice) is a member within the order Siphonostomatoida, the Caligidae.There are around 559 species in 37 genera, including approximately 162Lepeophtheirus and 268 Caligus species. While sea lice are presentwithin wild populations of salmon, sea lice infestations within farmedsalmon populations present especially significant challenges. Severalantiparasitic drugs have been developed for control purposes. L.salmonis is the major sea louse of concern in Norway. Caligusrogercresseyi has become a major parasite of concern on salmon farms inChile.

Sea lice have both free swimming (planktonic) and parasitic life stages.All stages are separated by moults. The development rate for L. salmonisfrom egg to adult varies from 17 to 72 days depending on temperature.Eggs hatch into nauplius I which moult to a second naupliar stage; bothnaupliar stages are non-feeding, depending on yolk reserves for energy,and adapted for swimming. The copepodid stage is the infectious stageand it searches for an appropriate host, likely by chemo- andmechanosensory clues.

Once attached to the host the copepodid stage begins feeding and beginsto develop into the first chalimus stage. Copepods and chalimus stageshave a developed gastrointestinal tract and feed on host mucus andtissues within range of their attachment. Pre-adult and adult sea lice,especially gravid females, are aggressive feeders, in some cases feedingon blood in addition to tissue and mucus.

The time and expense associated with mitigation efforts and fishmortality increase the cost of fish production by approximately 0.2EURO/kg. Accordingly, external fish parasites, such as sea lice are aprimary concern of contemporary salmon farmers, who dedicateconsiderable resources to averting infestations and complying withgovernment regulations aimed at averting broader ecological impacts.

Both effective mitigation (e.g. assessing the need and timing ofvaccination or chemical treatments) and regulatory compliance arereliant upon accurate quantification of external fish parasites, such assea lice populations within individual farming operations. Presently,counting external fish parasites, such as sea lice is a completelymanual and therefore time consuming process. For example, in Norway,counts must be performed and reported weekly, presenting an annualdirect cost of 24 M $ alone. Equally troublesome is the questionablevalidity of statistics based on manual counts, when a count of externalfish parasites, such as adult female sea lice on a sample of between 10and 20 sedated fish is extrapolated to determine appropriate treatmentfor populations of over 50,000 fish. Consequently, both over-treatmentand under-treatment are common.

WO 2017/068127 A1 describes a system of the type indicated in thepreamble of claim 1, aimed at enabling automated and accurate detectionand counting of external fish parasites, such as sea lice within fishpopulations.

Any such system based on optical imaging must overcome severalsubstantial challenges associated with marine environments and animalbehavior.

-   -   Optical distortion from density gradients. The turbulent mixing        of warm and cold water or, especially, salt and fresh water        (e.g. within fjords) generates small scale density variations        causing optical distortion. The impact upon the imaging of        objects of less than 1-3 mm (e.g. juvenile lice) is especially        severe.    -   Fish aversion to unfamiliar light sources. Fish may exhibit a        fear response or more general aversion to light sources of        unfamiliar location, intensity, or spectra. Distortion of fish        shoals around such a light source will generally increase the        typical imager-to-fish distance, decreasing the effective acuity        of the imaging system. The cited document addresses this problem        by providing a guide system for guiding the fish along a desired        imaging trajectory.    -   Focus tracking in highly dynamic, marine environments.        Commercially available focus-tracking systems do not perform        well in highly dynamic scenes in which a large number of quickly        moving, plausible focus targets (i.e. a school of swimming fish)        are concurrently present within the field of view.

It is an object of the invention to provide a system and methodaddressing these challenge and providing accurate automated counts inorder reduce the amount of human labor associated with external fishparasites, such as sea lice counts and enable more effective predictionand prevention of harmful infestations.

SUMMARY

In order to achieve this object, the system according to the inventionis characterized in that:

-   -   a ranging detector configured for detecting the presence of fish        and measuring a distance from the detector to the fish is        mounted adjacent to the camera, and    -   an electronic control system is arranged to control a focus of        the camera on the basis of the measured distance and to trigger        the camera when a fish has been detected within a predetermined        distance range.

Rather than attempting to guide the fish along a certain trajectory, theapproach according to the invention is to trigger the camera only when afish is detected within a suitable distance range. Thus, the fish areallowed to follow their natural flocking behavior, and the camera andthe ranging detector may be installed near a track which the fish arelikely to follow due to their natural behavior, e.g. along a boundary ofthe sea pen.

Since the system captures an image only when a fish has actually beendetected, the number of images to be captured and, accordingly, thenumber of images to be analyzed can be limited. Further, sinceillumination light is needed only at the time when an image is actuallytaken, irritation of the fish by illumination light sources is alsoreduced.

The capability of the ranging detector to provide reliable distance datais used to for control the focus of a camera more precisely and therebyenhances the quality of the captured images.

More specific optional features of the invention are indicated in thedependent claims.

Preferably, the system is able to detect and categorize external fishparasites, such as sea lice of both sexes at various sessile, mobile,and egg-laying life stages (e.g. juvenile, pre-adult, adult male, adultfemale egg-bearing, and adult female non-egg-bearing).

Furthermore, the system could form the basis of an integrated decisionsupport platform improving the operational performance, animal health,and sustainability of ocean-based aquaculture.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiment examples will now be described in conjunction with thedrawings, wherein:

FIG. 1 shows a side view of a camera and lighting rig according to apreferred embodiment of the invention;

FIG. 2 is a view of a sea pen with the rig according to FIG. 1 suspendedtherein;

FIG. 3 shows a front view of the camera and lighting rig;

FIG. 4 shows a side view of an angular field of view of a rangingdetector mounted on the rig;

FIGS. 5 and 6 are diagrams illustrating detection results of the rangingdetector;

FIGS. 7-10 show image frames illustrating several steps of an imagecapturing and analyzing procedure;

FIG. 11 shows a flow chart detailing a process of annotating images andtraining, validating, and testing an external fish parasite detectorwithin an electronic image processing system (machine vision system)according to an embodiment of the invention; and

FIG. 12 shows a flow chart detailing the operation of the external fishparasite detector in an inference mode.

DETAILED DESCRIPTION

Image Capture System

As shown in FIG. 1 , an image capture system comprises a camera andlighting rig 10 and a camera and lighting control system 12 enablingautomated acquisition of high-quality images of fish.

The camera and lighting rig 10 comprises a vertical support member 14,an upper boom 16, a lower boom 18, a camera housing 20, an upperlighting array 22, and a lower lighting array 24. The camera housing 20is attached to the vertical support member 14 and is preferablyadjustable in height. The vertical positioning of the camera ispreferably such that the field of view of the camera is at leastpartially (preferably mostly or entirely) covered by the lighting conesof the upper and lower lighting arrays 22, 24. Also, there is preferablya substantial angular offset between the centerline of the camera fieldof view and the centerlines of the lighting cones. This minimizes theamount of light backscattered (by particulates in the water) to thecamera, maximizing (relatively) the amount of light returned from thefish tissue. In the shown setup, the camera may be mounted at a height,as measured from the blower end of the support member 14, between ¼ and¾ of the length of the vertical support member.

The upper boom 16 and lower boom 18 couple with the vertical support 14member at elbow joints 26 and 28, respectively, that allow angulararticulation of the upper boom and lower boom relative to the verticalsupport member. The upper lighting array 22 and lower lighting array 24couple to the upper boom and lower boom at pivotable joints 30 and 32,respectively, that allow angular articulation of the upper lightingarray and lower lighting array relative to the upper boom and lowerboom.

In the example shown, suspension ropes 34 constitute a bifilarsuspension for the camera and lighting rig 10. The suspension ropespermit to control the posture of the rig in azimuth and can be attachedto a bracket 36 in different positions, thereby to keep the rig inbalance for the given configuration of the booms 16 and 18. This enablesfine adjustment of the orientation (i.e. pitch angle) of the camera andlighting rig as the center of mass of the camera and lighting rigcenters below the attachment point.

Preferably, a cabling conduit 38 carries all data and power required bythe upper lighting array, lower lighting array, and camera housingbetween the camera and lighting rig and the camera and lighting controlsystem 12.

FIG. 2 shows a diagram of the camera and lighting rig 10 immersed in asea pen 40. The exemplary sea pen shown is surrounded by a dock 42 fromwhich vertical support members 44 extend upward. Tensioned cables 46span between the support members. The suspension ropes 34 can attach tothe tensioned cables 46 to allow insertion and removal of the camera andlighting rig 10 into the sea pen as well as to control the horizontalposition of the rig relative to the dock 42.

It should be noted, however, that the sea pen may also have a shapedifferent from what is shown in FIG. 2 .

Extension of the supporting cables and ropes also allows adjustment ofthe depth of the camera and lighting rig below the water surface.Preferably, the camera and lighting rig is placed at a depth thatpositions the camera housing 20 below the surface mixing layers wherethe turbulent mixing of warm and cold water or salt and fresh water ismost pronounced. This further reduces the optical distortion associatedwith density gradients. The required depth varies based on location andseason, but typically a depth of 2-3 m is preferred.

As is shown in FIG. 3 , the upper lighting array 22 and lower lightingarray 24 comprise horizontal members 48 that support one or morelighting units 50 within a lighting array along their length. In theembodiment shown in FIG. 3 , the upper lighting array and lower lightingarray each comprise two lighting units 50, however, different numbers oflighting units may be used. The horizontal members 48 couple to theupper boom and lower boom at the pivotable joints 30, 32.

The elbow joints 26, 28 between the vertical support member 14 and upperboom 16 and lower boom 18 and the pivotable joints 30, 32 between theupper boom and lower boom and the horizontal members 48 collectivelyallow for independent adjustment of:

-   -   the horizontal offset between the camera housing 20 and the        upper lighting array 22,    -   the horizontal offset between the camera housing 20 and the        lower lighting array 24,    -   the angular orientation of the lighting units 50 within the        upper lighting array 22, and    -   the angular orientation of the lighting units 50 within the        lower lighting array 24.

Generally, the upper lighting array and lower lighting array arepositioned relative to the camera housing to provide adequate lightingwithin a target region where fish will be imaged for external fishparasites, such as sea lice detection. The lengthwise-vertical designand configuration of the camera and lighting rig 10 maximizes thelikelihood that fish (that exhibit aversion to long, horizontallyoriented objects) will swim in close proximity to the camera housing.Furthermore, the separate and independently adjustable upper lightingarray and lower lighting array allow for lighting schemes specificallydesigned to address lighting challenges unique to fish, as discussed ingreater detail below.

The camera housing 20, which is shown in a front view in FIG. 3 ,comprises a camera 52, a ranging detector 54, e.g. a light-basedtime-of-flight detection and ranging unit, and a posture sensing unit 56including for example a magnetometer and an inertial measurement unit(IMU) or other known posture sensing systems.

The camera is preferably a commercially available digital camera with ahigh-sensitivity, low-noise sensor, capable of capturing sharp images offast moving fish in relatively low lighting. In a preferred embodimentof the invention, a Raytrix C42i camera is used, providing a horizontalfield of view of approximately 60° and a vertical field of view ofapproximately 45°. Of course, any other camera with similar properties(including electronically controllable focus) may be used as analternative.

The ranging detector 54 is used to detect the range and bearing of fishswimming within the field of view of the camera 52. The detectorcomprises emit optics 58 and receive optics 60. The emit optics 58produce a fan of light oriented in the vertical direction but preferablycollimated in horizontal direction. That is, the fan diverges in pitch,parallel to the vertical support member, but diverges relatively littlein yaw, perpendicular to the vertical support member.

The receive optics 60 comprises an array of light detector elements,each detecting light incident from within an acceptance angle spanningat least a portion of the vertical field of view of the camera. Theangles of adjacent detector elements abut one another in pitch,collectively creating an acceptance fan that completely covers thevertical field of view. This orientation and configuration of the emitand receive optics is optimized to detect and locate the bodies of fish(which are generally high aspect ratio) swimming parallel to thehorizontal water surface.

Preferably, the ranging detector 54 operates on a wavelength of lightproviding efficient transmission within water. For example, blue lightor green light may be used to provide efficient transmission within seawater. In the preferred embodiment of the invention, the rangingdetector is a LEDDAR® detector such as LeddarTech M16, emitting andreceiving light at 465 nm. Of course, the invention is not limited tothis embodiment of a ranging detector.

Also in the preferred embodiment of the invention, the illumination fangenerated by the emit optics diverges approximately 45° in pitch,effectively spanning the vertical field of view of the camera, anddiverges approximately 7.5° in yaw. The receive optics 60 comprises andarray of 16 detector elements, each with a field of view spanningapproximately 3° in pitch and approximately 7.5° in yaw. Of course, thenumber of detector elements may be smaller or larger than 16, butpreferably not smaller than 4. Preferably, both the illumination fan andthe acceptance fan are horizontally centered within the camera field ofview, ensuring that detected fish can be completely captured by thecamera.

Systems with two or more ranging detectors may also be envisaged. Forexample, a fan could be positioned ‘upstream’ (as defined by prevailingdirection of fish swimming) of the centerline, to provide ‘advancedwarning’ of a fish entering the frame. Similar, a unit could be placeddownstream to confirm fish exiting the frame.

The IMU in the posture sensing unit 56 comprises an accelerometer andgyroscope, e.g. similar to those found in commercially available smartphones. In a preferred embodiment of the invention, the magnetometer andIMU are collocated on a single printed circuit board within the camerahousing 20. Collectively, the IMU and magnetometer measure theorientation of the camera housing (and therefore the imagery acquired bythe camera) relative to the water surface and the sea pen. Because fishgenerally swim parallel to the water surface and along the edges of thesea pen, this information can be used to inform a machine vision systemof an expected fish orientation within the acquired imagery.

The upper lighting array 22 and lower lighting array 24 can include oneor more lights of various types (e.g. incandescent, gas discharge, orLED) emitting light at any number of wavelengths. Preferably, thespecific types of lights are chosen to provide sufficient colorinformation (i.e. a broad enough emittance spectrum) for the externalfish parasites, such as sea lice to be adequately contrasted against thefish tissue. Additionally, the types and intensity of the lights withinthe upper lighting array and lower lighting array are preferablyselected to yield a relatively uniform intensity of light reflected tothe camera despite the typical, markedly countershaded bodies of thefish.

In the embodiment proposed here, the upper lighting array 22 comprises apair of xenon flashtubes. The lower lighting array 24 comprises a pairof LED lights, each comprising a chip with 128 white LED dies. Thishybrid lighting system provides a greater range of lighting intensitythan can be attained with a single lighting type. Specifically, theflashtubes provide brief but intense illumination (approximately 3400l×) from above the fish, synchronized to the operation of the camerashutter. This ensures adequate light reflected to the camera from thetypically dark, highly absorptive upper surfaces of the fish. (Thisrequires a greater intensity of light than could be delivered by the LEDlights of the lower lighting array.) Correspondingly, the LED lightsprovide an adequate lighting intensity for the typically light, highlyreflective lower surfaces of the fish. (This requires an intensity belowwhat could be provided by the xenon flashtubes of the upper lightingarray.) The resulting uniformly bright light reflected from the fishallows the camera to operate at a relatively low sensitivity (e.g. belowISO 3200) to provide low-noise images to the machine vision system.Finally, both the xenon flashtubes and LED lights provide an adequatelybroad spectrum to allow discrimination of the external fish parasites,such as sea lice from fish tissue.

As described above, the upper lighting array 22 and lower lighting array24 are positioned to provide the desired illumination across the targetregion. The target region is characterized by the vertical field of viewof the camera and near and far bounds along the axis of the camera. Thedistance from the camera to the near bound is the further of (a) theclosest attainable focus distance of the camera and (b) the distance atwhich a typical fish spans the entire horizontal viewing angle of thecamera. The distance from the camera to the far bound is the distance atwhich the angular resolution of the camera can no longer resolve thesmallest external fish parasites, such as sea lice that must bedetected. The near bound “a” and the far bound “b” are illustrated inFIG. 4 .

Each of the lights within the upper lighting array and lower lightingarray provide a generally axisymmetric illumination pattern. Becausethere are multiple lights within each array along the length of thehorizontal members, the illumination pattern can be effectivelycharacterized by an angular span in the pitch plane. The length of thevertical support member 14, the angular position of the upper boom 16and lower boom 18, and the angular orientation of the upper lightingarray 22 and lower lighting array 24 are preferably adjusted such thatthe angular span of the upper lighting array and lower lighting arrayeffectively cover the target region. The distance from the camera to the“sweet spot depends on the size of the fish to be monitored and may bein a range from 200 mm to 2000 mm for example. In the case of salmon,for example, a suitable value may be around 700 mm.

In practice, the intensity of the illumination provided by the upperlighting array and lower lighting array are not completely uniform overtheir angular span. The above approach, however, ensures that anacceptable amount of illumination is provided over the target region. Italso results in a “sweet spot” a short distance beyond the near boundwhere the angle of illumination between the upper lighting array, lowerlighting array, and camera are optimal. This results in the best litimages also providing the best angular resolution attainable by thecamera and suffering minimally from density gradient distortions.

A wide variety of other camera and lighting geometries may be utilizedwithout departing from the scope of the invention. In particular, thecamera and lighting rig may be constructed for and positioned inorientations other than the vertical orientation of FIG. 1 . Forexample, the camera and lighting rig may be oriented horizontally,parallel to the water surface. The camera and lighting rig may also beconstructed to maintain one or more cameras in fixed positions (relativeto the target area) other than those shown in FIG. 1 . Additionally,some embodiments of the invention may incorporate multiple camera andlighting rigs, e.g. two camera and lighting rigs symmetricallypositioned in front and in back of the target region, enablingsimultaneous capture of imagery on both sides of a single fish.

Camera and Lighting Control System

The camera and lighting control system 12 controls the operation of theimage capture system. The camera and lighting control system:

-   -   receives and analyzes data from the ranging detector 54 to        determine an appropriate camera focus distance,    -   controls the camera focus and shutter,    -   controls the timing of the illumination of the upper lighting        array 22 and the lower lighting array 24 relative to the shutter        of the camera 52, and    -   receives, analyzes, and stores image data and image metadata,        including ranging detector, magnetometer, and IMU measurements.

In the present embodiment, the camera and lighting control system 12comprises a computer 62 and a power control unit 64 that reside at a drylocation (e.g. the dock 42) physically proximal to the camera andlighting rig 10. In an alternative embodiments, at least a portion ofthe camera and lighting control system functionality provided by thecomputer is performed by an embedded system below the water surface(e.g. mounted to the vertical support member 14 or integrated within thecamera housing. Generally, the computer 62 includes device drivers foreach sensor within the camera and lighting rig 10. In particular, thecomputer includes device drivers for the camera 52, the ranging detector54, and the magnetometer and the IMU of the posture sensing unit 56. Thedevice drivers allow the computer to acquire measurement data from andsend control data to the associated sensors. In a preferred embodiment,measurement data and control data are passed between the devices andprocesses running on the computer as messages in the Robotic OperatingSystem (ROS). Data from the sensors (including the ranging detector)arrive at a frequency of 10 Hz, and measurements from the magnetometerand IMU arrive at 100 Hz. Each of the messages is logged to disk on thecomputer.

The computer 62 provides control signals to the power control unit 64and optionally receives diagnostic data from the power control unit. Thepower control unit provides power via the cabling 38 to the upperlighting array 22 and the lower lighting array 24. In the preferredembodiment, the power control unit receives 220 V AC power, which canpass directly to a charger for a capacitor bank for the xenon flashtubeswithin the upper lighting array 22 (when triggered). The power controlunit passes power to an underwater junction box (not shown) thattransforms the AC power to DC power (e.g. 36 V or 72 V) for the LEDlights within the upper lighting array 24.

A focus calculation process, executed by the computer 62, continuouslymonitors the ranging data to detect the presence and determine the rangeof fish within the target region. The ranging data consists of one ormore distances, for each detector element, from which light wasreflected back to the detector element from within its acceptance anglewithin the acceptance fan.

FIG. 4 shows a side view of the angular fields of view 66 of thedetector elements in the receive optics 60 within a ranging acceptancefan 68. As described above, the acceptance angles of adjacent detectorsabut one another in pitch to create the acceptance fan. FIG. 4 shows anarray of 16 detectors, each with a field of view spanning approximately3° in pitch.

FIG. 4 shows average pitch angles of the detectors within the rangingacceptance fan 68. Each average pitch angle is illustrated by acenterline 70 bisecting the field of view 66 of the correspondingdetector. The average pitch angle is the angle between the bisectingcenterline 70 and the centerline of the acceptance fan 68 as a whole,which is generally parallel to the optical axis of the camera 52.

FIG. 4 also shows a side view of the distances and average pitch anglesfor several detector elements occluded by fish 72, 74 within the rangingacceptance fan 68. Generally, the focus calculation process detects fishwhen several adjacent detector elements report similar distances. In thepreferred embodiment of the invention, a fish is detected when M or moreadjacent detector elements report similar distances d_(i). The number Mmay be in the range from 1 to ½ the total number of detectors (i.e. 8 inthis example). Specifically, the focus calculation process looks foradjacent sets of M or more adjacent distances d_(i) for which[max(d_(i))−min(d_(i))]≤W. M and W are parameters that can be adjustedby the operator of the image capture system, with W representing amaximum allowable thickness approximately corresponding to half thethickness of the largest fish that will be detected. Depending on thesize or age of the fish the parameters M and W are optimized for eachsystem or pen. For each such detection, the focus calculation computesthe mean distanceD=(1/M)Σ₁ ^(M) d _(i)and a mean bearingβ=(1/M)Σ₁ ^(M)β_(i)

where β_(i) are the average pitch angles of each of the adjacentdetector elements. The focus calculation process then returns the focusdistance D_(f)=D*cos β, which represents the distance from the camera tothe recommended focus plane along the optical axis of the camera.

Multiple distances may be reported by a single detector due to scatteredparticulates in the water or an object (e.g. a fish) that subtends onlya portion of the acceptance angle of the detector. In a practicalembodiment, in those instances where a single detector reports multipledistances, the focus calculation uses the furthest distance. Thisminimizes the number of false detections induced by particulates withinthe water. In the event that the multiple distances are actuallyassociated with two fish, one of which occludes only a portion of thedetector's acceptance angle, it is likely that neighboring detectorswill still successfully detect the partially occluding fish.

The fish presence and range information determined by the focuscalculation process can be used to control the image acquisition of thecamera. For example, images may be captured only when a fish is detectedwithin a predetermined distance of the “sweet spot” providing optimallighting. For example, if the “sweet spot” is at 700 mm, images may becaptured only when a fish is detected within a distance range from 600to 800 mm. Whenever images are captured, the camera and lighting controlsystem sets the focus distance of the camera to the most recent rangevalue determined by the focus calculation process.

In a preferred embodiment of the invention, the camera and lightingcontrol system continually triggers the camera to acquire images on aperiodic basis, for example, at a frequency of 4 Hz or more generally, afrequency between 2 and 10 Hz. The focus calculation process continuallyand periodically (e.g. at 10 Hz or, more generally, at 4 to 20 Hz)reports a current focus distance based on the most recent fish detectionand ranges, and the camera and lighting control system sets the focusdistance of the camera to the latest available focus distance.

When the camera shutter opens, the camera sends a synchronization signalto the camera and lighting control system 12, which is passed to thepower control unit 64. The power control unit illuminates the upperlighting array 22 and lower lighting array 24 synchronized with theshutter, to ensure proper illumination of the captured image. In thoseembodiments of the invention where the lights within the upper lightingarray or lower lighting array are not able to maintain a duty cycleequal to the camera (such as the xenon flashtubes of the preferredembodiment of the invention), the power control unit can also include alighting inhibitor process that continually assesses whether the powercontrol unit should illuminate the upper lighting array and lowerlighting array. In a preferred embodiment of the invention, illuminationis inhibited if either (a) the firing history of the xenon flashtubeswithin the upper lighting array is nearing their thermal limit or (b)the focus calculation process has not recently detected a fish andreported an updated range.

In a preferred embodiment of the invention, the less intense LED lightswithin the lower lighting array are illuminated for the duration of thecamera exposure. The length of the exposure is set at the minimum lengthrequired for the LED lights to provide adequate illumination. The flashlength of the xenon flashtubes in the upper lighting array is adjustedto provide balanced lighting given the countershading of a typical fish.

The intensity of the illumination provided by the LED lights within thelower lighting array is preferably great enough to provide a shortenough exposure to yield acceptably low motion blur within the capturedimages of swimming fish. In the preferred embodiment of the invention,the sensor within the camera (in particular its pixel count), the opticsof the camera (in particular the angular span of the field of view) andthe distance to the target region are chosen to ensure that (a) a fullfish can be captured within the field of view of the camera yet (b) evenjuvenile external fish parasites, such as juvenile sea lice can beadequately resolved. Providing 10 pixels across each 2 mm (comparable tothe size of a juvenile sea lice) at a target distance at which the 60°horizontal field of view of the camera spans the width of a typicaladult fish requires an angular pixel spacing of 7.6×10⁻³° per pixel. Forfish swimming at typical speed of 0.2 m/sec, sub-pixel motion blur isensured with shutter times of less than 0.6×10⁻³ s. To deliveradequately low-noise imagery to the machine vision system, a sensor gainof less than ISO 3200 is preferred. This in turn requires illuminationof approximately 3000 lux across the target region.

FIG. 5 illustrates the results that would be obtained with the rangingdetector 54 in the situation depicted in FIG. 4 . What has been shownare the detection results of detector elements with the fields of viewhaving center lines ranging from +6° to −15°. Each black dot in FIG. 5represents a detection event where reflected light has been received bythe pertinent detector element. The position of the dot in the directionof the d-axis represents the distance of the detected object ascalculated from the run time of the light signal from the emit optics 58to the object and back to the receive optic 60.

As has been described before, the fish 72 and 74 are represented bydetections at approximately the same distance d1 and d2, respectively,for a number of adjacent detectors. For each individual detector, thedistance of the fish is the largest among the distances measured by thatdetector. The dots at smaller distances represent noise caused by smallparticulate matter in the acceptance fan.

In the situation illustrated in FIGS. 4 and 5 , the fish 74 is partlyobscured by the fish 72, so that an image of the entire silhouette ofthe fish can be obtained only for the fish 72 at the smaller distanced1. Consequently, the focus of the camera will be adjusted to thatdistance d1.

FIG. 6 is a time diagram showing the detections at the distance d1 as afunction of time t. It can be seen that the detections obtained from thefish 72, for angles β ranging from −3° to −15°, are stable over anextended period of time corresponding to the time which it takes thefish to swim through the acceptance fan 68. Consequently, the noisemight also be filtered-out by requiring that the detection is stableover a certain minimum time interval or, equivalently, by integratingthe signal received from each detector element over a certain time andthen thresholding the integration result.

In principle, a detection history of the type illustrated in FIG. 6might also be used for optimizing the time interval in which the camera52 takes a sequence of pictures, in order to assure that, on the onehand, the number of pictures does not become unreasonably large and, onthe other hand, that the sequence of pictures includes at least onepicture in which the entire fish is within the field of view of thecamera. For example, as shown in FIG. 6 , a timer may be started at atime t1 when a certain number of adjacent detector elements (three)detect an object that could be a fish. Then, the camera may be triggeredwith a certain delay, at a time t2, to start with taking a sequence ofpictures, and the sequence will be stopped at the latest at a time t3when the detector elements indicate that the tail end of the fish isleaving the acceptance fan.

FIG. 7 shows a field of view 76 of the camera at the time t1 in FIG. 6 ,when the nose of the fish 72 has just crossed the acceptance fan 68.

FIG. 8 shows an image captured by the camera 52 at a time somewhat laterthan t2 in FIG. 6 , when the entire silhouette of the fish 72 is withinthe field of view 76. At that instant, it can be inferred from thedetection results of the detector elements at β=−3° to −15° in FIG. 6that the center line of the fish will be at β=−9°, as shown in FIG. 8 .This information can be passed-on to the image processing system and mayhelp to recognize the contour of the fish in the captured image.

Returning to FIG. 1 , the computer 62 of the camera and lighting controlsystem 12 is connected to an image processing system 78 which has accessto a database 80 via a data management system 82.

Data Management System

The automated system for detecting and counting external fish parasites,such as sea lice also includes the data management system 82 whichincludes interfaces supporting the acquisition, storage, search,retrieval, and distribution of image data, image metadata, imageannotations, and the detection data created upon operation of the imageprocessing system 78.

Data Store

The data management system 82 receives imagery from the image capturesystem, for example, in the form of ROS “bags”. The data managementsystem unpacks each bag into, for example, a JPEG or PNG image and JSON(JavaScript Object Notation) metadata. Each of the JPEG images is storedwithin a data store.

Database

The JSON metadata unpacked from each ROS bag is stored within thedatabase 80 associated with the data store. Generally, the metadatadescribes the image capture parameters of the associated JPEG or PNGimage. For example, the metadata includes an indication of the centroidpixel location within the silhouette of the fish (e.g. the pixelcentered horizontally within the image, longitudinal center line of thefish) detected by the LEDDAR unit. This pixel location may optionally beused by the image processing system (described in more detail below) tofacilitate the detection of fish within the image.

The database 80 also stores annotation data created during an annotationprocess for training the image processing system 78 (described ingreater detail below). The database additionally stores informationcharacterizing the location, size, and type of fish and external fishparasites, such as sea lice detected by the machine vision system.Finally, the database stores authentication credentials enabling usersto log in to the various interfaces (e.g. the annotation interface orthe end-user interface) via an authentication module.

Image Processing System

In a certain embodiment, the invention uses the image processing system78 to perform the task of external fish parasites, such as sea licedetection. In the preferred embodiment of the invention, separate neuralnets are trained to provide a fish detector 84 and an external fishparasite detector 86. The fish detector first 84 detects individual fishwithin imagery acquired by the image capture system. The external fishparasite detector 86 then detects individual external fish parasites,such as sea lice (if present) on the surface of each detected fish.Preferably, the external fish parasite detector also classifies the sexand life stage of each detected louse.

The detectors are trained via a machine learning procedure that ingestsa corpus of human-annotated images. Use of a neural net obviates theneed to explicitly define the characteristics (e.g. extent, shape,brightness, color, or texture) of fish or external fish parasites, suchas sea lice, but instead draws directly upon the knowledge of the humanannotators as encoded within the corpus of annotated images.

FIG. 9 shows the position and silhouette of the fish 72 in the field ofview 76, as detected by the fish detector 84. The other fish 74 shown inFIGS. 4, 7 and 8 has been excluded from consideration in this embodimentbecause it is partly occluded by the fish 72. In a modified embodiment,it would be possible, however, to detect also the fish 74 and to searchfor external fish parasites, such as sea lice on the skin of the fish74, as far as it is visible.

In one embodiment of the invention, the depth of focus of the camera 52has been selected such that a sharp image is obtained for the entiresilhouette of the fish 72. In a modified embodiment, as shown in FIG. 9, the silhouette of the fish, as recognized by the fish detector, issegmented into sub-areas 88 which differ in their distance from thecamera 52. The distances in the different sub-areas 88 are calculated onthe basis of the ranging result obtained from the ranging detector 54.The distance values obtained by the various detector elements of theranging detector reflect already the effect of the angular deviationbetween the center line 70 of the field of view and the optical axis ofthe camera in the pitch direction. Further, for each point within thesilhouette of the fish 72, the effect of the angular deviation inhorizontal direction can be inferred from the position of the pixel onthe fish in the field of view 76. Optionally, another distancecorrection may be made for the relief of the body of the fish inhorizontal direction, which relief is at least roughly known for thespecies of fish in consideration.

Then, when a series of images is taken from the fish 72 (e.g. with afrequency of 4 Hz as described above), the focus may be varied fromimage to image so that the focus is respectively adapted to one of thesub-areas 88 in FIG. 9 . This permits to obtain high resolution imagesof all sub-areas 88 of the fish with reduced depth of focus and,accordingly, with an aperture setting of the camera which requires lessillumination light intensity.

FIG. 10 shows a normalized image of the fish 72 that is eventuallysubmitted to the external fish parasite detector 86. This image mayoptionally be composed of several images of the sub-areas 88 capturedwith different camera focus. Further, the image shown in FIG. 10 may benormalized in size to a standard size, which facilitates comparison ofthe captured image of the fish with the annotated images.

It will be observed that the image of the fish 72 as recognized in FIG.9 may be subject to distortion (horizontal compression) if theorientation of the fish is not at right angles to the optical axis ofthe camera. The normalization process resulting in the silhouette of thefish as shown in FIG. 10 may compensate for this distortion.

Further, FIG. 10 illustrates an optional embodiment in which thesilhouette of the fish has been segmented into different regions 90, 92and 94-100. The regions 90 and 92 allow the image processing system todistinguish between the top side and the bottom side of the fish forwhich, on the one hand, the skin color of the fish will be differentand, on the other hand, the illumination intensities and spectraprovided by the upper and lower lighting arrays 22 and 24 will bedifferent. Knowledge of the region 90 or 92 where the pixel on the fishis located makes it easier for the external fish parasite detector 86 tosearch for characteristic features in the contrast between external fishparasites, such as sea lice and the fish tissue.

The further regions 94-100 shown in this example designate selectedanatomic features of the fish which correlate with characteristicpopulation densities of the external fish parasites, such as sea lice ofdifferent species on the fish. The same anatomic regions 94-100 willalso be identified on the annotated images used for machine learning.This allows the external fish parasite detector to be trained orconfigured such that confidence levels for the detection of externalfish parasites, such as sea lice are adapted to the region that ispresently under inspection.

Moreover, when the external fish parasite detector 86 is operated in theinference mode, it is possible to provide separate statistics for thedifferent regions 94-100 on the fish, which may provide usefulinformation for identifying the species, sex and/or life stage ofexternal fish parasites, such as sea lice and/or extent of infestation.

Annotation, Training, Validation, and Testing

FIG. 11 shows a flow chart detailing the annotation of images and thetraining, validation, and testing of the detectors 84, 86 within theimage processing system 78. The annotation and training process beginswith the acquisition of images. An annotation interface 102 allowshumans to create a set of annotations that, when associated with thecorresponding images, yields a corpus of annotated images.

In the preferred embodiment of the invention, the annotation interfacecommunicates with a media server that connects to the datastore in thedatabase 80. The annotation interface may be HTML-based, allowing thehuman annotators to load, view, and annotate images within a webbrowser. For each image, the annotator creates a polygon enclosing eachprominently visible fish, and a rectangular bounding box enclosing anyexternal fish parasites, such as sea lice present on the surface of thefish. Preferably, the annotation interface also allows the annotator tocreate rectangular bounding boxes enclosing fish eyes (which may bevisually similar to external fish parasites, such as sea lice).Preferably, the annotator also indicates the species, sex, and lifestage of each louse.

The annotations created using the annotation interface 102 are storedwithin the database 80. Upon insertion and retrieval from the database,the annotations for a single image are serialized as a JSON object witha pointer to the associated image. This eases the ingest of theannotated corpus by the machine learning procedure.

In the preferred embodiment of the invention, the machine learningprocedure comprises training neural nets on the corpus of annotatedimages. As shown in FIG. 11 , the annotated images may be divided intothree sets of images. The first two sets of images are used to train andvalidate a neural network. More specifically, the first set of images(e.g. approximately 80% of the annotated images) is used to iterativelyadjust the weights within the neural network. Periodically (i.e. after acertain number of additional iterations) the second set of images(approximately 10% of the annotated images) are used to validate theevolving detector, guarding against over-fitting. The result of thetraining and concurrent validation process is a trained detector 84, 86.The third set of images (e.g. approximately 10% of the annotated images)is used to test the trained detector. The testing procedurecharacterizes the performance of the trained detector, resulting in aset of performance metrics 104.

As shown in FIG. 11 , the entire training, validation, and testingprocess may be iterated multiple times, as part of a broader neutralnetwork design process, until acceptable performance metrics areattained. As noted above, in the preferred embodiment of the invention,the process of FIG. 11 is performed at least once to produce the fishdetector 84 and at least once to produce the external fish parasitedetector 86.

In alternative embodiments of the invention, to improve the quality ofthe training, validation, and testing process, the machine learningprocedure includes a data augmentation process to increase the size ofthe annotated corpus. For example, applying augmentation techniques suchas noise addition and perspective transformation to the human-annotatedcorpus can increase the size of the training corpus by as much as afactor of 64.

The invention claimed is:
 1. A system for automated external fishparasites monitoring in aquaculture, the system comprising: a camera(52) suitable to be submerged in a sea pen (40) suitable for containingfish (72, 74), the camera being arranged for capturing images of thefish; and an automated electronic image processing system (78)configured for identifying external fish parasites on the fish byanalyzing the captured images, characterized in that: a ranging detector(54) configured for detecting the presence of fish and measuring adistance from the detector to the fish is mounted adjacent to the camera(52); and an electronic control system (12) is arranged to control afocus of the camera (52) on the basis of the measured distance and totrigger the camera (52) when a fish has been detected within apredetermined distance range in which the distance is larger than thedistance at which a typical fish spans the entire horizontal viewingangle of the camera and smaller than the distance at which the angularresolution of the camera can no longer resolve the smallest externalfish parasites.
 2. The system according to claim 1, wherein the rangingdetector (54) comprises: emit optics (58) for emitting light; andreceive optics (60) adapted to receive reflected light and to measure atime of flight of the received light.
 3. The system according to claim2, wherein the emit optics (58) is configured to emit light in the formof a fan spread over an extended angular range in vertical direction andcollimated in horizontal direction.
 4. The system according to claim 3,wherein the receive optics (60) includes a plurality of detectorelements having adjacent fields of view (66) which collectively form avertically oriented acceptance fan (68), the ranging detector beingadapted to measure bearing angles (□) under which reflected light isdetected by the individual detector elements.
 5. The system according toclaim 1, wherein the electronic image processing system (78) comprises afish detector (84) configured to recognize a silhouette of a fish in thecaptured image, and an external fish parasite detector (86) configuredto detect external fish parasites in a specified region (90-100) withinthe silhouette of the fish (72).
 6. The system according to claim 5,wherein the electronic image processing system (78) comprises at leastone neural network constituting the fish detector (84) and/or theexternal fish parasite detector (86).
 7. The system according to claim1, comprising a camera and lighting rig (10) having a vertical supportmember (14), an upper boom (16) articulated to a top end of the supportmember (14) and carrying an upper lighting array (22), a lower boom (18)articulated to a lower end of the support member (14) and carrying alower lighting array (24), and a housing (20) attached to the supportmember and carrying the camera (52) and the ranging detector (54),wherein the upper and lower lighting arrays (22, 44) are configured toilluminate, from above and from below, a target region inside a field ofview of the camera (52).
 8. The system according to claim 7, wherein theupper lighting array (22) is configured to emit light with an intensityand/or spectral composition different from that of the light emitted bythe lower lighting array (24).
 9. The system according to claim 8,wherein the upper lighting array (22) comprises a flash tube unit, andthe lower lighting array (24) comprises an LED lighting unit.
 10. Thesystem according to claim 7, comprising a posture sensing unit (56)adapted to detect a posture of the camera and lighting rig (10) relativeto the sea pen (40).
 11. The system according to claim 1, wherein theelectronic control system (12) is configured to control the camera (52)so as to capture a sequence of images in an extended time interval inwhich the ranging detector (54) continuously detects the fish within thepredetermined distance range.
 12. A method for external fish parasitesmonitoring in aquaculture, characterized by using the system accordingto claim
 1. 13. A method for automated external fish parasitesmonitoring in aquaculture, comprising the steps of: submerging a camera(52) in a sea pen (40) comprising fish (72, 74); capturing images of thefish (72, 74) with the camera (52); and identifying external fishparasites on the fish (72, 74) by automated electronic image analysis ofthe captured images, characterized by the steps of: operating a rangingdetector (54) to continuously monitor a part of the sea pen (40) fordetecting the presence of fish in that part of the sea pen and, when afish has been detected, measuring a distance from the camera (52) to thefish (72, 74); and when a fish has been detected, calculating a focussetting of the camera (52) on the basis of the measured distance; andtriggering the camera (72) when the detected fish (72, 74) is within apredetermined distance range in which the distance is larger than thedistance at which a typical fish spans the entire horizontal viewingangle of the camera and smaller than the distance at which the angularresolution of the camera can no longer resolve the smallest externalfish parasites.
 14. The method according to claim 13, wherein theelectronic image processing system (78) is used for detecting fish (72,74) in an image captured by the camera (52) and for detecting externalfish parasites within a silhouette of the detected fish.
 15. The methodaccording to claim 14, wherein the ranging detector (54) is used formeasuring a bearing angle of a detected fish (72), and the measuredbearing angle is used in the machine vision system (78) for searchingfor the silhouette of the fish in the captured image.
 16. The methodaccording to claim 13, wherein a target region bounded by a field ofview of the camera (52) and by said predetermined distance range isilluminated from above and below with light of different intensitiesand/or spectral compositions.
 17. The method according to claim 16,wherein the step of detecting the presence of external fish parasites ata given location on the fish includes distinguishing whether said givenlocation is in a top side region (90) or a bottom side region (92) ofthe fish.
 18. The method according to claim 13, comprising:distinguishing between at least two different classes of external fishparasites which differ in the difficulty of recognizing the externalfish parasites; calculating quality metrics for each captured image, thequality metrics permitting to identify the classes of external fishparasites for which the quality of the image is sufficient for externalfish parasites detection; and establishing separate detection rates foreach class of external fish parasites, each detection rate being basedonly on images the quality of which, as described by the qualitymetrics, was sufficient for detecting external fish parasites of thatclass.
 19. A method for external fish parasites monitoring inaquaculture, characterized by using the system according to claim
 5. 20.A method for external fish parasites monitoring in aquaculture,characterized by using the system according to claim 7.